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This Appeal is ftom a 
referred to as the * Tinal Actioi: 



I Lnal< 



a Notice of Appeal and Pre- 
A4)pellants hereby submit this 



Office Action mailed on November 
*) finally rejecting claims 1-37. This 

Brief Request for Review filed on 
Appeal Brief in fiirtheiance of the Appeal 



Ap seal 



L - REAL PARTY IN IN ^REST 



The real party m inteieflt 
Machines Coq)oration, the assjjgnee 
application by virtue of an 



assi prnient 



XL RELATED APPEALS ; AND INTERFERENCES 



c • 



There are no Appeals 
the Assignee, which would dirj^tly 
Board's decision in the pendinlt Appeal. 



in. STATUS OF CLAIM S 



Claims 1 -37 are pendii g. 
the attached Appendix. Claim ; 
diiectly or indirectly from bas< 
claim 19. Claims 29-37 deper i 



29, 2005 (hereinafter, 
/jippeal was commenced by 
April 11,2006, and 



for the above-identified application 
of the entire right, title and 
of recorded in the U.S. Patent 



interest 



International Business 
in and to the subject 
ind Trademark Office. 



Interferences known to Applicant, Applicant 
aflfect or be indirectly affected by 



stand rejected and are under appeal. 
1, 19 and 29 are independent claims 
claim 1. Claims 20-28 depend directly 
directly or indirectly fiK>m base cLaini 



icant's representatives or 
or have a bearing on the 



The claims are set forth in 
Claims 2-18 depend 
or indirectly from base 
29. 



1 
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IV. STATUS OF AME^fD ^NTS 



No After Final Amendn ents have been filed. 



V, SUMMARY OF CLA MED SUBJECT MATTER 



In general^ the claimed 
multi-modal user interfaces (U 
for enabling synchronized mull 
authored to have multiple modi 
application in one, some or all 
the multi-modal application, 
described with reference to 
(hereinafter, " Spec.' "), for e: 
scope of the invention. 

A system for synchronMng 



F>r 



I car am 



3xai ipl 



a program storage dev^ 
application comprising at leas ' 
application in a first modality 
the application in a second mi 

a program execution 
synchronizes the first and secAnd mode prod 
application, wherein the prog} am 

a multi^nwdal shell thi t 
mode processes of the multi^r^dal 



L with 



aventions are directed to systems and 
) and ^plications and, more particulaijly 
-modal user interaction with a multimpdal 
processes to allow user interaction 
if interaction modes supported by the 
purposes of illustration, the claimed 
Figures and corresponding text 
e, but nothing herein shall be 



deem< ;d 



enables 



n, the muUirmodal 
' interaction with the 
user interaction with 



multi-modal interactions, comprising: 
thai stores a multi-modal applicati on, 
a first mode process that enables us$r } 
%nd a second mode process thai ^ 
daliiy; 

sj^stem that executes the multimodal i 

esses while a user inter^ts i 
execution system comprises: 
manages information exchanges bStween the first and second 
application to enable a synchroiiized multi^modal 



ipethods for implementing 
, to system and methods 

application that is 
the multimodal 
c^ifferent mode processes of 
inventions will be 
of j^ppellants* Specification 
as a limitation on the 



application and 

with the multi'modal 
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interaction with the multi-mot al application wherein user interactic n 
in execution of corresponding commands in both the first and second 

an API (application pr tgram interface) that allows the first 
to register their respective acti ^e commands and corresponding 



shell. 



An exemplary embodin lent of the invention of claim 1 is 
corresponding tejct of Appellar Ls Spec. In particular, FIG. 1 illustrate 

I interactiom (10) (see generally, pace 



for synchronizing multi-modal 



interaction with the applicatici 



while a user interacts with the 



in one modality results 
mode processes; and 
^nd second mode processes 
actibns with the multi-modal 



illusirated by FIG. 1 and 

an exemplary system (10) 
1. line 3 -- p. 15. line 8Y 



The system includes program s torage device that stores a multi-moda 
multi-modal application(12) c >mprising at least a first mode process 

(12) in a first modality (e.g., speech 
second mode process (12b) ths t enables user interaction with the 
modalhy (e.g., GUI interactior modality).(see, e.g.. Spec., p. 11, line 
lines 3-16). 

The system (1 0) includlfcs a program execution system (18, 20 
multi-modal application (12) e id synchronizes the first and second 

I nulti-modal application (12) (Spec. p. 
program execution system cor iprises a multi-modal shell (20) that 
exchanges between the first ar i second mode processes (12a, 12b) 
application (12) to enable a sy ichronized multi-modal interaction 
application (12) wherein user nteraction in one modality results in 
commands in both the first an| second mode processes (see. Spec. p. 
The program execution systenl comprises an API (application prograjm 



application (12) , the 
(12a) that enables user 
interaction modality) and a 
application (12) in a second 
18 -p. 12, line l;page 14, 



22) that executes the 
rtiode processes (12a, 12b) 
14, lines 8-16). The 
mages information 
of the multi-modal 

the multi-modal 
e:^ecution of corresponding 
11, line 18 --p. 12, line 6). 
interface) (18) that 



wih 
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allows the first and second mode processes (12a, 12b) to register their 

commands and corresponding j ctions with the multi-modal shell (20) 

20- p. 14. line 2; p. 19, lines sjllS) 

Claim 2 recites; 
The system of claim /, 



comprises a list of each of the 



respective active 

(see, e.g., Spec. p. 12, line 



therein the program execution systeht comprises a registry 
having a registration table, mi naged by the multi-modal shell, wherein the registration table 

' actions 
execulfion of a registered 



registered commands and correspond ing synchronized 



that results in both the first ai d second mode processes upon 
command by one of the first a td second mode processes. 



FIGs. 1 and 2 illustrate 
(20). As Ulustrated in FIG. 2, 
the registered commands and 
and second mode processes (1 
the first and second mode 
21, Une 7 ~ p. 22, line 7). By 
considered a "two-way* 
GUI. In the illustrative 
results in both the speech mod 
application 12 for a given 



Claim 12 recites! 

The system of claim I, 
(40) comprising a first browsSr 
e.g., no. 4, pp. 24-27 of Spe«i.) 



regiitered 



a registration table (28) that is managejd 
or example, the registration table (28) 
I corresponding synchronized actions (2 > 
a) and (12b) upon execution of a 

(12a, 12b) (see, e.g.. Spec. p. 13 
|vay of example, the registration table ! 
table because of the two 
of Fig. 1, the registration table ^ 
3 process 12a and the GUI mode procefss 
in one modality. 



proc ssses 



regist ation 



emboc iment 



i exemplary 



command 



wherein the multi-modal application i? 
application (41) and a second brows er 



by the multi-modal shell 
comprises a list of each of 
that results in both the furst 

command by one of 
line 10 p. 14, line 2; p. 
8 ofFig. 1 maybe 

modalities, speech and 
8 indicates what action 
12b of the multi-modal 



a multi-modal browser 
application (42). (See, 



4 



PACE 11/34 * RCVD AT 10/12/2006 4:01:16 PM [Eastern Daylight TImel * 8VR:USPTO-EFXRF-6/38 * DNI8:2738300 ' C8ID:516 602 8889 * DURATION (mm^s):11-54 



Oct 12 2006 4:02Pri F.CHHU & RSSOCIflTES, LLC 516-692-8889 



p. 12 



Claim 13 recites: 

The system of claim 12, 
browser is a speech browser 



wherein the first browser is a GUI 
(1 ig. 4 illustrates a GUI and speech 



t \e 



Claim 19 recites; 

A method for synchronMng 
2 and 3, and generally, page 21 

activating a multi-modi\l 
enables user interaction with 
that enables user interaction h 

receiving a command 

23); 

triggering an action by 
event (element 3, FIG. 2, page 
process (12a); and (see, 

updating application 
processes (element 3, FIG, 2, 



eleme its 



si ites 



de\ ^ce 



Claim 29 recites: 

A program storage 
instructions executable by the 
interactions, the method 

activating a mtdti-modlkl 
enables tdser interaction with 
that enables user interaction 

receiving a command <{Jr 

23); 
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bro\^ser and the second 
browser 41 and 42) 



multi-modal interactions, comprigfng the steps of (Sec FIGs. 
line 7 23, line 17): 
application comprising at least afir^ 
application in a first modality and a 
th the application in a second modality, 
event in the first modality (element 1 



the received command or 
by the second mode 



the first mode process (12a) based on 
and triggering a corresponding actior 
2 and 3, FIG. 2, p, 22, lines 1-5) 
or device states associated with t!\e first and second mode 
22, lines 5-7). 



readable by a machine, 
nachine to perform method steps for 
the steps of: 
application comprising at least a 
application in a first modality and 
ith the application in a second modality, 
event in the first modality (element l|. FIG. 2, p, 21 , lines 20- 



comp ising 



i He 



1 first 



mode process [Viz) that 
second mode process (12b) 



FIG. 2, p. 21, lines 20- 



tangibly erhbodying a program of 

s,/nchronizing multi- modal 



mode process (12b) that 
second mode process (12b) 



5 
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triggering an action by 
event (element 3, FIG. 2, page 
process (12a); and (see, 

updating application 
processes (element 3, FIG. 2, p 



, elemei ts 



sttvtes 



he first mode process (12a) based on 
I jnd triggering a corresponding action 
2 and 3, FIG. 2» p, 22, lines 1-5) 
or device states associated with th^ first and second mode 
22, lines 5-7). 



VI. GROUNDS OF REJECTION TO BE REVIEWED ON AF PEAL 



A. Claims 1-15 anic 
over U.S. Patent No. 6,1 19,1 4^ 

B, Claims 16-18 
Hashimoto in view of U.S. 



ai ; 
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1 9-37 are rejected under 35 U.S.C. lC(3(a> as being mpalentable 
to Hashimoto 

rejected under 35 U.S.C. 103(a) as b^ing unpatentable over 
Patent No. 6,119, 147 to Toomev . 



VI* ARGUMENTS 
A. The Obviottsness Reidctions Based on Hashimoto arc Legailly Deficient 



"In rejecting claims 
presenting a prima fiicie case 
1955, 1956 (Fed. Cir. 1993) 
(Fed. Cir. 1992)). It is well 
general, a two part analysis - 
and substance of the subject 
whether the claimed subject 

More specifically, wheji 
the broadest reasonable i 



1445 



un<8er 35 U.S.C. Section 103, the examinejr 
c[|r obviousness." In re Rijckaert, 9 F.3d 
(cjlting In re Oetiker, 977 F.2d 1443, 
es^bblished that a prima facie showing of 
^(arting with a claim interpretation anal; 
mifrtter being claimed, followed by an 
matter (as interpreted) is obvious in vie>^ 
evaluating claims, the Examiner mus|t 
interpljetation that is consistent with the 



received command or 
by the second mode 



bears the initial burden of 
1531, 1532, 28 USPQ2d 
,24 USPQ2d 1443, 1444 
obviousness requires, in 
iVsis to determine the scope 
obvious analysis to determine 
of the prior art. 
interpret the claims with 
specification and the Examiner 
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langi^ge 



whe n 



cannot intetpret the claim 
view of the specification as it 
Sneed> 710 R2d 1544, 1548, 2 
833, 15 USPQ2d 1566, 1567 
1023, 1027 (Fed. Cir, 1997). 
has the burden of establishing 
obviousness is established 
suggested the claimed subject 
781,783, 26USPQ2dl529, 
189 USPQ 143, 147 CCCPA 

It is respectfully 
legally deficient to establish a 
below, the obviousness 
which the Examiner fails to 
the claimed inventions by pi 
effort to meet the claim 
the claim language nor proper 
rejections &il to set forth a 
the inventions of claims 1,19 

(1) Claim 1 is 



(led. 



in a vacuum, but rather the claims 
4ould be interpreted by one of ordinary 
8 USPQ 385, 388 (Fed. Cir. 1983); h\ 
Cir. 1990); In re Morris. 127 F,3d 
Once the claims have been properly 
^prima facie case of obviousness. 

the teachings from the prior art 
ij|xatter to a person of ordinary skill in 
1 (Fed. Cir. 1993) (quoting /« re Ri 



lt3 



1S76)). 



submit »d 



languai^e 



In general, Hashimoto 
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should be interpreted in 
skill in the . See, In re 
reBond,910F.2d 831, 



that the obviousness rejections set 
J jrima facie case of obviousness. Indeec , as will be explained 
rejecti()|ns are seemingly based on an improper obviousness analysis in 
properly interpret the claimed subject matter, but merely reconstructs 
pick||ng and choosing among diflFerent teacl lings of Hashimoto in an 
with no due consideration given to 1 he scope and meaning of 
tontext to the teachings o f Hashimoto, 



1048, 1054,44USPQ2d 
c|)nstructed, the Examiner 
'A jirima facie case of 
itselflwould appear to have 

exirinreBelU 991 F.2d 
Rinkhart, 531 F.2d 1048, 1051, 



tie 



brth in the Final Action are 



legilly sufficient reasoned argument as to 

i^r 29 as a whole 
PatAitable and Non Obvious over Hashimoto 



\ i s directed to speech recognition interfiice systems that can be used 
simultaneously by a plurality c f application programs to realize a spe 3ch I/O modality for 



in this regard, the 
how Hashimoto s uggests 
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applications in windows-based lystems (see, Col. 3, lines 28-35). Has limoto teaches speech 



keyboard and mouse inputs anc 
voice commands to the task in 



ei;sent 



interfaces for GUI based opexat ng systems using speech recognition 
and command and controL Th< se speech applications, however, are 
GUI based operating systems ii the sense that such applications allow 

allow a user to change the focus, launph 
bcus. la such applications, the 
considered merely a new addit^nal I/O modality rather than a separat^ 
multimodal application. 

In this respect, at the veSy least, Appellant contends that 



a ^plications for dictation 
tially add-ons to the 
for replacement of 
new taslts, and give 

of Hashimoto is 
process mode of a 



speech intedGace 



Hashimoto does not does not 



in I 



0Q1 



teach or suggest a multi-modal application comprising at least a first 
user interaction with the applU ation in a first modality and a second 
user interaction with the applii ation in a second modality, as recited 
fundamental flaw in the Exami ier*s obviousness rejections is based 
understand and properly deterr tine the scope of the subject matter rej 
application. In accordance wit l the claimed inventions, a multimodal 
and second mode processes is xroperly construed as an application 
modalities (e.g., the applicatio] 1 12 having the speech and GUI modes 
only one of the modalities is pi esent (e.g., a GUI modality), the conmi 
missing modalities will not be u^tivated. An advantage, therefore, 
modal shell 20 is that the appli ^tion developer can program a multi 



including, in the first instance. 



all the information related to the diffeient 



the application, and then rely c a the multi-modal shell platform to mi inage. 



mode process that enables 
node process that enables 
claim I. One 
the Examiner's failure to 
^^arding a multi-modal 

application having first 
bitten for multiple 

12a and 12b), where when 
ands registered for the 
of Implementing a multi- 
modal application by 

modalities supported by 
, control and 
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modality specifi : rendering and synchronization when 



IS). 



t obvioi jness 



0 



11 tes 



: compr ses 



FKJs 



i s 



implement the 

(see, e.g.. Spec. p. 14, lines 3- 

The Examiner's 
the Final Action. In an effort 
application, the Examiner cites 
midti-modal appUcc^ion fonne[p 
application programs (2), and 
corresponding text (Col. 41, li 
multimodal application 
cites other embodiments in 
49, lines 50-63, for example) 
multimodal application enable 
second modality. 

On a fundamental leve 
Examiner are very different 
Hashimoto as teaching a **mu]j|i 
recognition system) 1 and a p 
misplaced. FIG. 6 discloses 
functions on behalf of one or 
commands using recognition 
application program (2) carri^ 
the received recognition resul 
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in interaction with the user. 



IS : 



analysis with respect to claim 1 i 
show how Hashimoto discloses the 
one embodiment of FIG. FIG. 6 of Ha si 
by the combination of a SRS (speech 
1 len cites another embodiment in FIG. 
23-25 and Col. 42, lines 37-38) as 
a first mode process and a second 
. 1 7 and 66 of Hashimoto ( and correp 
disclosing that the first and second 
user interaction with the multimodal 



set forth on pag^s 2-3 of 
4laimed multi-modal 
himoto as teaching a 
recognitions system) and 
56 of Hashimoto , and 
l eaching that the 
mode process, and then 
iponding description in Col. 
niode processes of the 
^plication in a first and 



Examiner 



, however, Hashimoto's user interface 
the claimed inventions. The 
modal application" formed by the 
jrality of application programs (2) is 

embodiment where the SRS (1) perfc 
nore different applications (2) by 
rocabularies provided by each of the 
out its own processing depending on 
(see, e.g. Col. 10, lines 26-375 Col 



a 1 



methods as cited by the 
's reliance on FIG. 6 of 
c<imbination of a SRS (speech 
unclear, and in any event 
xprms speech recognition 
input speech 
►plications (2), Each 
the application, according to 
2, Unes 23-30). 



procsssmg 



a>i 
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of 



■reli ^ 



if 



In this regard, FIG, 6 
the claimed invention. At 
(2) ia a speech modality (spee^ 
system, not a "mode process 
event rely on the embodiment 
mode processes, but rather 

In particular, FIG. 56 
system (1) of FIG. 6 is extended 
from the application program 
teaching first and second mod 
Action). However, at most, F 
interaction mode, i.e., speech 

Notwithstanding this, 
second mode processes, but 
processes" of FIG. 56 dopQt 
modality. Instead, the Examiiiler 



not disclose a "multi modal applicajti^ 
FIG. 6 describes user interaction 
commands). The SRS (1) is merely 
a given multimodal application. In 
>f FIG. 6 to show a multi-modal 
on the embodiment disclosed by Ha 
Hashimoto is an extended embodim<i3t 



CoL 49, lines 50-63 and FIG. 
modalities. 

FIG. 17 illustrates a 
used to control a plurality o 
multi-modal ii^ut/output witl 
application (5) is a multi-modtu 



BEST AV/JiUBLE COPV 



ion" as contemplated by 
with ^dch separate application 
a speech recognition 
^act, the Examiner does not 
applidadon with first and second 
shimoto in FIG. 56, 



:(14) 



ExJiminer ; 



to include a speech synthesis unit 
I 2) (see. Col. 41, lines 11-27). The 
processes of a multimodal application 
&. 56 illustrates an embodiment for i 

Examiner erroneously relies on FIG 
gly acknowledges that the supposed 
4^1ow user intemction with the system ih 
instead relies on other embodiments i 
7, 66 and 69-72 to show user interaction 



of FIG. 6 where the SRS 
to enable speech outputs 

reUes on this as 
(see page 2 of the Final 
inlplementing only one user 



lie: 



s€ ^nunc 



sp »ch recognition system (3) and a wi 
f a jplications (5). Although Hashimoto 



an ^plication (5), Hashimoto does 
application with separate first and 



10 



56 as teaching first and 

*first and second mode 
a first and second 
disclosed by Hashimoto in 
in first and second 



wm<jows system (4) that can be 
Arguably teaches in FIG. 1 7 
n6t d isclose that the 
second mode processes. In 
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1). 



Morec ven 



mes£ iges. 



submit ed that 



fiact, the speech system (3) 
plurality of Explications (coi 
multimodal applicatiotL 
speech mail tooL Although thi 
(speech synthesis) e-mail 
multi-modal , 

Therefore, it is 
Hashimoto discloses a multi- 
enables user interaction with 
enables user interaction with 
and (652) in FIG. 66 are not 
contemplated by the claimed 
separate mode processes of a 
inventions. 

It is further submitted 



moce 



andllwiiidow system (4) simply operate the 
lines 4- 16), but are not separate 
, FIG. 66 illustrates an example 
system allows multi-modal I/O 
, or generate e-mails by voice 



n odal i 



i ie . 



di Serent i 



mode processes 



a program 
and synchronizes 
with the multimodal 
comprises: 

a muWr^modal^heU 
first and second 
synchronized multi-mibdal 
tiser interaction in on * 
commands in both thi first 

an API (applic^^itiOi 
mode processes to 
actions with the multimodal shell 



BEST AVAIUBLE COPY 



I/O devices shared by a 
processes of a 
em)x>diment of FIG- 17 for a 
fimctil^nality display or read out 
input) or keyboard. 



(sp<jech i 



the Examiner has not sufficiency 
application comprising at least 
application in a first modality and 
application in a second modality. 
mode processes of a multi-moc^al 
invention. Likewise, I/O system (3) anc 1 
fiiulti-modal application, as contemplate 



1 aat Hashimoto does not disclose or suj ^est 



exe 'ution . 



system thai executes the 
the jhrst and second mode processes whil£ 
i pplication, wherein the program 



that manages information 

of the multi^modal 
interaction with the 
modality results in execution of cori[i 
and second mode processes; 
n program interface) that allows 
their respecth^e active commands 



ami 



II 



p. 18 



demonstrated how 
first mode process that 
second mode process that 
^Lgain, the I/O systems (65 1) 
application, as 
(4) of FIG. 16 are not 
by the claimed 



multt-fnodal application 
a user interacts 
execution system 



exchanges 



between the 
applicaiion to enable a 
multi^modal application wherein 
esponding 



ifte first and second 
and corresponding 
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In an attempt to establis h 
parsing the claim language into 
various teachings of Hashimot< 



a|;ain 



leading to inconsistency and 

In particular, as set fort l 
embodiment in FIG. 56 



these claim features, the Examiner 
variolas segments in a way that allows 
Is disclosure to fit to the claim language 
in|oherency in the analysis. 

on Page 3 of the Final Action, the E: 
of HasMimoto contending that the MPU (: 



the claimed ^multi-modal shellt 
example, as support for Hashiiiioto 



exchanges between the first am 
a synchronized multi-modal i 
in one modality results in 
mode processes. The Examinei 
In the fust instance, as 
multimodal application have 
In HGs. 6 and 56, the MPU (1 
whole by exchanging 
for decoding of speech input 
speech output to the user, to 
10, lines 38-48; Col. 41, lines 
merely provides a platform 
but there is no suggestion that 



ml taction ' 



cxeci ition 



mode processes of a multi-moc al application. 
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Cmessage 



, but relies on a different embodiment 
teaching that the mtdti-modal shell 



bothi 



second mode pn>cesses of the multi 

with the multi-modal applicatjon 
of corresponding commands in 
s analysis is way off target and highly 
( iscussed above, the embodiment 

process modes synchronized thi 
) controls the functions of the speech 
the speech recognition xmit 
a user and for controlling the speec|i 
enltble speech I/O with each of the a] 

-34). In this regard, the speech I/O 
for [Implementing speech FO interface 
MPU (\ V\ manages information 



Si karate i 



messages between 



)m I 



:7- 



1 rte] 



12 



offers an analysis by 
the Examiner to cite 
in each parsed segment. 



x$mmer relies on the 

processing unit) (1 1) is 
disclosed in FIG. 66, for 
manages information 
ijiodal application to enable 
wherein user interaction 
the first and second 
inconsistent, 
of FIlQ-, 56 does not disclose a 
ough a mialti-modal shell, 
ifecognition unit (12) as a 
tiie application programs 
synthesis unit (1 4) for 
pplicjations (2), (see, e.g.. Col. 
5|ystem(lA)ofFIG. 56 

of the applications (2), 
exchanges between separate 



aidi 



of tiach^ 
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Moreover, FIO. 66 
(65 1) similar to the speech I/O 
addition to a windows system 
speech I/O system (651) is 
regard, there is nothing in 



illustrates an exemplary embodiment havi 
system (1 A) in FIG. 56 (to provide a 
I 652) (to provide a GUI modality) 
siniU^ to that depicted in HG. 56 (see, 
Has ttdmoto t hat suggests the MPU (1 1) in 



seei lingly i 



manages information exchangfji 

Therefore, there is 
the speech recognition system 
information exchanges betwe^ 
explication to enable a 
claim 1. 

Moreover, the flaw in 
characterization of the messa^ 
(application program interface) 
their respective active commiinds 
FIGs. 6 and 1 1 , the message 
program (22) to exchange 

The Examiner previoij^ly 
application** formed by the 
application programs (2). So 
shell, where tiie speech I/O i 
second mode process (2) to 



s between different mode processes 

no basis for Examiner's 
[1 A) as being a multi-modal shell for 
the first and second mode processes 
synch^nized multi-modal interaction with 



BEST AVi\ILABLE 



COPY 

haviilg a speech I/Q system 
speech I/O modality) in 
Haihimoto teaches that the 



Cd 



tie I 



.47 lines 18-22). In this 
speech I/O system (651) 
o^ the application (653). 
characterization of the MPU (1 1) of 
Managing and synchronizing 
7f the multi-modal 
application^ as recited in 



th^ 



he Examiner's analysis is further evi 
I/O unit (21) of each application (2) 
that allows the first and second 

and corresponding actions with 
unit (21) of an application (2) 

with the MPU (1 1) of the SRS ([l). 
contended that FIG- 6 of Hashim< 



10 



me! sages 



enable 



ith<j 



coHnbination of a SRS (speech recognitiop 
vhat the Examiner now contends is 
(21) is an API that allows the 
raster their respective active commands 



in er&ce i 



13 



evidenced by the Examiner's 
)eing essentially an API 
mc de processes to register 
the multi-modal shelL In 
allows the application 



loto teaches a '*multi-modal 
system) 1 and a plurality of 
MPU (1 1) is a multimodal 
(SRS) mode process and 
and corresponding actions 



first 
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withtbeiMPU(ll), However, 
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the flaw with this argument is that Hashimoto does act remotely 



teach or suggest that the API (|l) of the application prog^Mi (2) provides means allowing the 
SRS (1) (purported "mode process") to register its conunands with tJi3 MPU (1 1) (purported 
multi-modal shell") 

For at l east above reas()}ns, the teachings of Hashimoto as citeft by the Examiner are 

a prima facie case of obviousness aga nst claim 1 . 
and Nog Obvioas over Hashimoto 



legally insufficient to establish 
2. Claim 2 is Patentable 



Appellant submit that he Examiner has feiled to show how J ashimoto teaches or 



suggests B, program execution 



. On page 4 of the Final 
met by Hashimoto^s teaching 



system comprises a registry having a 



managed by the muld-modal ^hell, wherein ike registration table c emprises a list of each of 
the registered commands ontAeorresponding synchronized actions that results in both the first 
and second mode processes u\7on execution of a registered command by one of the fmt and 
second mode processes 

Action, the Examiner contends that th 5 subject matter of claim 2 is 
[ n FIG. 56 of a program management table (13) as being the 
claimed ^'registration table" lich is managed by the MPU (11) (wh ich Examiner interprets as 
the claimed "multi-modal she 1")- Although Hashimoto arguably teaches that the PMT (1 3) - 
stares a recognition vocabulai y for a speech input and a destination application (2) for sending 
the recognition result (see, Cc 1. 10, lines 48-57), there is nothing in 

of each of the registered commands md corresponding 
synchronized actions that rei utts in both the first and second modi* processes upon execution 
of a registered command by < ne of the first and second mode proc isses. Indeed, as explained 



14 



the PMT (13) comprises a lis 



registration table. 



Hashimoto that suggests that 
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above, the PMT (13) at most st )res data related to implement a speeci interface for each 
application (2). It is simply be^ /ildering how fte Examiner could iBte pret this PMT (13) as 
storing lists of registered comn ands and corresponding synchronized 
process modes of an applicatio i, and the explanation given on page 4 



actions for two separate 
in this regard is 



unintelligible. 

3. Claims 12 and 13 are |*atentable and Non Obvious over itashimoto 



AppeUant submit that t le Exammer has failed to show how H ashimoto teaches that the 



multi-modal application is a n ulti-modal browser comprising a first 



second browser application^ a 
GUI browser and the second 
On page 6 of the Final 
a multimodal application (1 A) 



recited in claim 12, much less wherein the first browser is a 
i browser is a speech browser, as recitec in claim 13. 
Action, Ac Examiner contends that FIG. 56 of Hashimoto teaches 
is a multi-modal browser (mail browasr, in FIG, 27) comprising 



LABLE COPY 



browser application and a 



first and second browser app]i :ations (multiple application programs (2) in FIG. 7). 

Again, the basis for thi > fmding is confusing and unintelligible. It is untenable to 
characterize the speech I/O sy tem (lA) of FIG. 56 as being a "multi modal Bqpplication*', much 
less a multi modal browser co nprising first and second browser applications. At the very least, a 
mail tool client (e.g., Microso t Outlook) is not considered a browser application (e.g.j Internet 
Explorer, etc.) to one of ordin ny skill in the art. 

Moreover, the Examir sr^s argument seems to suggest that th 5 SRS system (1 A) 
comprises a first and second \ rowscr appUcation (as provided by m\]ltiple applications (2)). At 
the very least, FIG. 6 disclose i that the application programs (2) are 



carry out its own processing c spending on its application. There is i imply no basis for the 



15 



separate programs which 
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Hashimoto for similar reasons 



BEST 
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Examiner to contend that Hashi cnoto teaches the applications (2) in Fip. 6 are first and second 
browser ^plications of a multi nodal browser application. 

4, Claims 19 and 19 are Patentable and Non Obviou s <[>ver Hashimoto 



Appellant respectfully s ibmit that claims 1 9 and 29 are patent ible and non-obvious over 

jive above for claim 1 , For instance, ^ [ashimoto does not disclose 



or suggest activating a multi-n odal explication comprising at least c 
enables user interaction with t te application in a first modality and a 
enables user interaction with t ie application in a second modality , as 
29. The crux of the Examiner 3 rejection of claims 19 and 29 is simi 

above claim limitations. Acco dingly, Appellants previous argument^; for claim 1 are applicable 
in this regard. 

The remaining argume t its set forth by the Examiner are unint( slligible For example, the 
Examiner states on page 5 of t le Final Action that Hashimoto teache 3: 

receiving a coi imand (speech input, line 14, cclum 1 41) in a first 
modality (speech reci gnition interface system, line 13 column. 41); 

triggering an j ction (speech output, line 19, columi 41) in the first 
mode process based < n the received command (speech recognition interface 
system, line 13 colum 1 14) and triggering a correspondini; action (the mail is 
opened when the use saying •'yes, lines 54-60 column 25) 
process (application irogram, Unes 41-42 column 42). 



The flaws with this 
only a Speech I/O modality (1 
based on a strained parsing 



first mode process that 
second mode process that 
recited in claims 19 and 
ar to claim 1 at least for the 



ar] ^uxlent 



s n- 



at d 



are that the Examiner relies on 
gle interaction modality) and that the 
interpretation of the claim language 



16 



by the second mode 



FIG. 56 which illustrates 
Examiner's aa^ument is 
as applied to different 
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speei h 



Exaj liner 



embodiments in Hashimoto . 
Examiner^s belief that the 
modal application, andthethaj 
multimodal application, which p 
as discussed above. 

Most tellingly, the 
application state associated ^vi1b 
TTajghiTn^to does not teach 
second mode processes, 
modalities, the application 
synchronization to maintain 
dialog. 

Accordingly, for at 
prima facie case of obvioiisnei|s 
B- The Obviousness Rei 



n\oc 



Mi^reover, the Examiner's argument is 
recognition system provides a. first 
application programs provide a secorui 
an improper interpretation as appliec 



Indec i, 



1 Stat ;s 



ith 5 



:Iea: t 



intc] action 



Further, with respect t<; 
Hashimoto and Toomev^ sucli 
extent that Hashimoto does n()|t 
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eatoneously based on the 
\de process of a multi- 
mode process of the 
to the claimed inventions 



»dces 



; underscores 



iner acknowledges that Hashimoto 
thie first mode process. This 

with a multi-modal application |^1h 
when intereu^ting with a multimodal 
for each of the mode processes musi 
separate mode processes in the proper 



not disclose updating 
the fact that 
synchronized first and 
application in two 
be updated in 
current context of the 



the above reasons, the Final Office 
against clauns 1, 2, 12, 13 19 and 29 
etlons Based on Hashimoto and 



\ fith 



the rejection of claim 16-1 8 based on 
combination is believed to be legally 
disclose or suggest the inventions 
no elaboration necessary, Toomev 



16-18 depend. Furthermore, 

deficiencies of Hashimoto wi|i regard to claim 1 as discussed abov<! 
C. Conclusion 



i^ction fails to present a 
based on Hashimoto . 
Todmev are Legallv Deficient 



the combination of 
leficient at least to the 
of Iclaim 1, finom which claims 
clearly does not cure the 



17 
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Accordingly, for at least 
reverse all claim rejections unde ' 



be above reasons, it is respectfully requested that the Board 
35 U.S.C. §103. 

RjBspectfiilly subnjitted 



F. Chan & Associates, LLC 
130 Woodbury Road 
Woodbury, New York 1 1797 
TEL: (516) 692-8888 
FAX: (516)692-8889 
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Frank V. DeRosa 
Reg. No. 43,584 
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' sy icJ 



: devi « 



Claims Appendix 

1. A system for 

a program storage 
application comprising at least 
application in a first modality 
application in a second modali|y; 

a program execution 
the first and second mode 
\^erein the program executioi 

a multi-modal shell 
mode processes of the multi-i 
with the multi-modal appl; 
corresponding commands in 

an API (application 
register their respective active 



hronizing multi-modal interactions, (tomprismg 
that stores a multi-modal applicatioi|, 
a first mode process that enables user 
) nd a second mode process that enables 



s> stem 1 



proc ^sses ^ 



[th£ t 



licati m 



b>th1 



pn gram : 



2. The system 
registry having a registration 
table comprises a list of each 
actions that results in both the 
command by one of the first 



3. The system of 
first mono-mode application 
the second mode process, 
exchanges between the first 



4. The system 
interface modality and a 



RECEIVED 

BEST AVAIUBLE C0P^^^^^''^^^'^^^fl 

OCT 1 2 2006 



the multi-modal 
i nteraction with the 
user interaction with the 



thai executes the multi-modal 
while a user interacts with the 
system comprises: 

manages information exchanges bet>^ 
nilodal application to enable a synchroni: :ed 
wherein user interaction in one mod ality 
the first and second mode processe j 
interface) that allows the first £m|d 
cormnands and corresponding actions 



api>lication and synchronizes 
mylti-modal application. 



eenthe first and second 
multi-modal interaction 
results in execution of 
;and 

second mode processes to 
with the multi-modal shell. 



of jilaim 1, wherein the program execution 
t ible, managed by the multi-modal shel 
< f the registered commands and corresi5>onding 
first and second mode processes upon 
aiul second mode processes. 



;laim 1, wherein the multi-modal application 
the first mode process and a 

the multi-modal shell manages 
second mono-mode applications. 



wh( rein 



a id 



of Maim 1, fiirther comprising at least a 
secoiid device having a second user int^rfac 



19 



system comprises a 
, wherein the registration 

synchronized 
execution of a registered 



comprises at least a 
application for 
synchronizes information 



second mono-mode 



I and 



first 



device having a first user 
e modality, wherein the 
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multi-modal shell manages and synchronizes information exchanges letween the first and second 
devices. 



5 . The system of claim 
shell are distributed over a netirork, 
or protocols. 



6. The system of i 
mono-mode application to a 



7. The system o 
and wherein the mechanism 
the active commands of the 
commands to support the 



(4aim 2, wherein the API comprises a it|echanism for converdng a 
i- modal application. 



m jlti- 



f 



aim 6, wherein the mono-mode 
pij[)vides speech enablement of the GUI j 
application and building a grammar 
in a speech modality. 



QUI 



com aands 



8. The system of c laim 2, wherein the API comprises a mechanism for building a 
multi-modal application. 



9, The system 
the registry by building a 
actions for each of the 



10. The system 
modal shell executes on top o! 



12. TThe system 
browser comprising a first 



4, wherein the first device, second 
and wherein the API is i 



device and multi-modal 
implemerited using distributed APIs 



of ^laim 8, wherein the mechanism is use<^ 
regiAration table having user-defined 
modalitpes of the multi-modal application. 



applicjation is a GUI application, 
pplicalion by registering 
for the registered 



of ftlaim 1, further comprising an operatiii|g system, wherein the multi- 
the operating system. 



1 1 - The system of : jlaim 1 , wherein the system is distribii ed over a network. 



of itlaim 1, wherein the multi-modal 
br^lwser application and a second browser 



20 



for directly programming 
comiiiands and corresponding 



application is a multi-modal 
application. 



PACE 27/34 ■ RCVD AT 10J12/200B 4:01:16 PM [Eastern Daylight TimeJ • SVR:USPTO-EFXRP-6/36 " DrilS:2738300 * CSID:! 



:516 692 88S9 " DURATION (mm-ss): 11-54 



Oct 12 2006 4:08PM F,CHRU & RSSOCIRTES, LLC 516-692-8889 



p. 28 



14. The system 
application to send modality-! 



13. The system of qfaim 1 2, wherein the first browser is a pUl browser and the 
second browser is a speech brdwser. 



of 1, aim 



12, wherein the multi-modal shell 
specific presentation information to the 



15. The system of 
modality-independent descripflon 
specific presentation informat}pn 



<jjlaim 14, wherein the mvdti-modal 
and wherein the multi-modal shell 
fi-om the modality-independent description, 



16. The system 
combination of declarative 



17. The system of ol 
declarative markup languages and 
synchronization between the j|rst 



18. The system of 
files of the declarative markup 
the separate files are loosely 



of ^laim 14, wherein the mxilti-modal 
mAknp languages to describe the first anA 



-mo< al 



19. A method for 
activating a multi-i 
user interaction with the 
user interaction with the 

receiving a conmiand 
triggering an action hi 
and triggering a correspondin j 



BEST AVAILABLE 



processes the multi-modal 
Respective browsers. 



appllication is authored using a 
genemtes the modality- 



appication 



aim 16, wherein the multi-modal app 
synchronization elements in singk 
and second mode processes. 



j;laim 1 6, wherein the multi-modal appllication 
languages for the first and second mo|de 
sjtnchronized at predefined points. 



^(ynchronizing multi-modal interactions 
Explication coraprisii^ at least a firpt 
appHbation in a first modalit>' and a second 
appi cation in a second modality 
event in the first modahty ; 
the first mode process based on the relceived command or event 
action by the second mode process; {[nd 



i)r 1 



21 



COPY 



comprises a 
second mode processes. 



ication combines the 
pages to provide tight 



comprises a separate 
processes and wherein 



, comprising the steps of: 

mode process that enables 
mode process that enables 
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updating application i 
processes. 



sti tes or device states associated with the 



20. The method 

registering active 
associated with the second 

associating, with each 
mode process and a 

associating, with each 
second mode process and a 



comr lands 



mole 



icorrespon mg 



of ttlaim 19, further comprising the steps of: 

associated with the first mode process and active commands 
process; 

ijegistered command of the first mode 

action on the second mode proces^ 
jegistered command of the second mode 
corresponding action on the first mode prbcess 



21. The method 
action registration table based 



of Itlaim 20, further comprising the step of building a command-to- 
3n the registered conunands and actioi is. 



22. The method of 
shell via API calls from the 
processes. 



23. The method of 
looking up the receive|^ 
executing the actions 
processes. 



24. The method 
registering a callback 
second mode processes 



25, The method 
device states comprises the 



BEST available: copy 



process, an action on the first 
; and 

process, an action on the 



laim 21 , wherein the registration tabl<! 
aplblications or devices associated with tl|ie 



slaim 20, wherein the step of triggering comprises the steps of: 
command in the registration table; aqd 
dissociated with the received command 



ofllclaim 20, further comprising the steps 
[andle for each of the registered commlands 
of coi I ipletion of the actions corresponding tt* 



offlclaim 24, wherein the step of updating; 
sttos of executing the callback handle 
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first and second mode 



is built by a multi-modal 
first and second mode 



Dy the first and second mode 



of: 



to notify the first and 
the registered commands. 



the application states or the 
associated with the received 
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command to trigger a callback 
second mode process. 
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! fcction on the first mode process and a (tailback action on the 



26. The method of 
executing first thread 
triggering a oorrespondj|ng 
second mode process. 



< :laim 19, wherein the step of triggering 
associated with the received command; 
second thread to initiate the c 



storge 



27, The method of claim 26, wherein the threads are applets 

28. The method of jflaiin 26, wherein the threads conrmnm^cate via socket 
connections. 



tJ e 



29. A program 
of instructions executable by 
modal interactions, the metho< 

activating a multi-modftl 
user interaction with the applii 
user interaction with the application 

receiving a command 

triggering an action by 
and triggering a correspondiD^; 

updating application 
processes. 



comprises the steps of: 
ind 

action by the 



orresponding 



tangibly embodying a program 
synchronizing multi- 



for 



cr 



St ites 



mode process that enables 
that enables 



device readable by a machme, 
machine to perform method steps 
comprising the steps of: 
application comprising at least a firsjt 
ation in a first modality and a 
in a second modality 
event in the first modality; 
the first mode process based on the 
action by tiie second mode process; 

or device states associated with th|s first and second mode 



30, The program sfprage device of claim 29, further comi]>rising instructions for 
performing the steps of: 

registering active comihandi 
associated with the second m^lde 



s associated with the first mode p|rocess and active commands 
process; 
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second mode process 



received command or event 
ajid 
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espoiM ing 



associating, with each 
mode process and a corr< 

associating, with each 
second mode process and a coi 



^gistered command of the first mode 

action on the second mode process 
gistered command of the second mod s 
] jresponding action on die first mode pn)cesa 



31. The program 
performing the step of building ; 
commands and actions. 



St xrage 



32. The program 
the step of triggering comprisej 
Looking up the receiveii 
executing the actions 
processes. 



stjjjrage device of claim 30, wherein the iistructions for performing 
instructions for performing the steps cf: 
command in the registration table; anji 
altsociated with the received command tty the first and second mode 



; of con pletii 



si >rage 



34, The program 
the step of updating the appliclktii 
performing the step of executi ig 
trigger a callback action on tin 
process. 



35. The program 
the step of triggering compris^ 
. executing first thread 



BEST AVAIL«VBLE COPY 



p rocess, an action on the first 
and 

process, an action on the 



device of claim 30, fiirther comptising 
a command-tO'-action registration tabl 



instructions for 
e based on the registered 



33. The program stjprage device of claim 30, fiirther compjrising instructions for 
performing the steps of: 

registering a callback 
second mode processes 



[ conimjuids 



Ijpndle for each of the registered 

on of the actions corresponding tc* 



device of claim 33, wherein the 
on states or the device states compri 
the callback handle associated with 
first mode process and a callback 



action 



st xrage i 



device of claim 29, wherein the Instructions for performing 
instruction for performing the steps < 
dissociated with the received command; 
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to iiotify the first and 
the registered commands. 



^stmctions for performing 

instructions for 
the received command to 
on the second mode 



(|>f: 
and 
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triggering a oorresponc ivg second thread to initiate the con:es])onding action by the 
second mode process. 



36. The program st )rage device of claim 35, wherein the threads are applets. 

37, The program si )rage device of claim 35, wherein the threads communicate via 
socket connections. 
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Evtdeace Appendix 

There is no evidence siBbmii 
evidence entered by the exaroijfcer 
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itted pursuant to 37 CFR§§ 1,130, 
and relied upon by appellant in this 



RECEIVED 
CENTRAL FAX CENTER 



OCT 1 2 2005 

L131 or 1.132 or any other 
Appeal. 



26 



PACE 33/34 ' RCVD AT 10/12/2006 4:01:16 PM [Eastern Daylight Time]" SVR:USPTO-EPXRF-6/38 * DNIS:2738300 • CSID:i 



;516 692 8889 « DURATION (mm-ss): 11-54 



Oct 12 2006 4: 10Pri F.CHRU & HSSOCIRTES, LLC 51S-692-8889 



p, 34 



Related Proceedings Append x 



None. 
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